Statistical principal components analysis for retrieval experiments

© 2007 Wiley Periodicals, Inc. • Published online 22 January 2007 in Wiley InterScience (www.interscience.wiley.com). DOI: 10.1002/asi.20537 three fundamental components: a set of documents, a set of posed information needs, and a set of relevance judgments. Relevance judgments are the collections of documents that should be retrieved for each information need, and a posed information need is a query that may be formulated by any inquirer (user). In this paradigm, relevance is the sole effectiveness criterion and the effectiveness of a system based on relevance is measured in two dimensions: the ability to retrieve documents that are known to be relevant and the ability to suppress documents that are known to be nonrelevant. The majority of the currently used measures of relevance are based on precision and recall. Precision is the proportion of retrieved documents that are relevant; recall is the proportion of relevant documents that are retrieved. This experimental design paradigm has been in use for over four decades, and it is still actively used in almost all large-scale experimental evaluation efforts. In the traditional evaluation of retrieval experiments, performances of the systems are measured over a set of queries (or information needs, or topics).1 Because a performance summary measure is necessary to compare different retrieval strategies over all predefined information needs, a final summary performance score for each retrieval strategy is calculated as the average of its performance scores observed on all topics. In particular, the mean average precision (MAP) is the most widely used summary measure. A MAP score of a particular retrieval strategy is the mean of the uninterpolated average precisions observed on all topics, and in turn, an uninterpolated average precision score of a document set retrieved by a particular retrieval strategy is the average of all precision scores that are calculated at each relevant document reached from the beginning in that document set (van Rijsbergen, 1979). For visual performance comparisons, a recall-precision graph can also be used (van Rijsbergen, 1979). To construct a recall precision graph for a number of different retrieval strategies, all individual recall-precision curves of those Statistical Principal Components Analysis for Retrieval Experiments